{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "B905bc8x4-I6"
      },
      "source": [
        "#### Licensed under the Apache License, Version 2.0 (the \"License\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "YcK9kdXi4q8l"
      },
      "outputs": [],
      "source": [
        "# @title Copyright 2019 The TensorFlow Authors.\n",
        "\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xu2SVpFJjmJr"
      },
      "source": [
        "# DeepDreaming with TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "AE4X-5Z6qJlv"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/community/en/r1/deepdream.ipynb\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003e\n",
        "    Run in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/examples/master/tensorflow/examples/blob/master/community/en/r1/deepdream.ipynb\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003e\n",
        "    View source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rx1FbclXm8c7"
      },
      "source": [
        "\u003e For a TensorFlow 2.0 compatible \n",
        "version see [TensorFlow.org](https://tensorflow.org/en/beta/tutorials/generative/deepdream.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "toc",
        "id": "hupz2hrZjdnC"
      },
      "source": [
        "\u003e[Loading and displaying the model graph](#loading)\n",
        "\n",
        "\u003e[Naive feature visualization](#naive)\n",
        "\n",
        "\u003e[Multiscale image generation](#multiscale)\n",
        "\n",
        "\u003e[Laplacian Pyramid Gradient Normalization](#laplacian)\n",
        "\n",
        "\u003e[Playing with feature visualzations](#playing)\n",
        "\n",
        "\u003e[DeepDream](#deepdream)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-PLC9SvcQgkG"
      },
      "source": [
        "This notebook demonstrates a number of Convolutional Neural Network image generation techniques implemented with TensorFlow for fun and science:\n",
        "\n",
        "- visualize individual feature channels and their combinations to explore the space of patterns learned by the neural network (see [GoogLeNet](https://storage.googleapis.com/deepdream/visualz/tensorflow_inception/index.html) and [VGG16](https://storage.googleapis.com/deepdream/visualz/vgg16/index.html) galleries)\n",
        "- embed TensorBoard graph visualizations into Jupyter notebooks\n",
        "- produce high-resolution images with tiled computation ([example](https://storage.googleapis.com/deepdream/pilatus_flowers.jpg))\n",
        "- use Laplacian Pyramid Gradient Normalization to produce smooth and colorful visuals at low cost\n",
        "- generate DeepDream-like images with TensorFlow (DogSlugs included)\n",
        "\n",
        "\n",
        "The network under examination is the [GoogLeNet architecture](http://arxiv.org/abs/1409.4842), trained to classify images into one of 1000 categories of the [ImageNet](http://image-net.org/) dataset. It consists of a set of layers that apply a sequence of transformations to the input image. The parameters of these transformations were determined during the training process by a variant of gradient descent algorithm. The internal image representations may seem obscure, but it is possible to visualize and interpret them. In this notebook we are going to present a few tricks that allow to make these visualizations both efficient to generate and even beautiful. Impatient readers can start with exploring the full galleries of images generated by the method described here for [GoogLeNet](https://storage.googleapis.com/deepdream/visualz/tensorflow_inception/index.html) and [VGG16](https://storage.googleapis.com/deepdream/visualz/vgg16/index.html) architectures."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "jtD9nb-2QgkY",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "# boilerplate code\n",
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "import os\n",
        "from io import BytesIO\n",
        "import numpy as np\n",
        "from functools import partial\n",
        "import PIL.Image\n",
        "from IPython.display import clear_output, Image, display, HTML\n",
        "\n",
        "import tensorflow.compat.v1 as tf\n",
        "tf.disable_v2_behavior()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ILvNKvMvc2n5"
      },
      "source": [
        "\u003ca id='loading'\u003e\u003c/a\u003e\n",
        "## Loading and displaying the model graph\n",
        "\n",
        "The pretrained network can be downloaded [here](https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip). Unpack the `tensorflow_inception_graph.pb` file from the archive and set its path to `model_fn` variable. Alternatively you can uncomment and run the following cell to download the network:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "9ozsAvJdn2G7",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "!wget -nc  https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip \u0026\u0026 unzip -n inception5h.zip"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "1kJuJRLiQgkg",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "model_fn = 'tensorflow_inception_graph.pb'\n",
        "\n",
        "# creating TensorFlow session and loading the model\n",
        "graph = tf.Graph()\n",
        "sess = tf.InteractiveSession(graph=graph)\n",
        "with tf.gfile.FastGFile(model_fn, 'rb') as f:\n",
        "    graph_def = tf.GraphDef()\n",
        "    graph_def.ParseFromString(f.read())\n",
        "t_input = tf.placeholder(np.float32, name='input') # define the input tensor\n",
        "imagenet_mean = 117.0\n",
        "t_preprocessed = tf.expand_dims(t_input-imagenet_mean, 0)\n",
        "tf.import_graph_def(graph_def, {'input':t_preprocessed})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "eJZVMSmiQgkp"
      },
      "source": [
        "To take a glimpse into the kinds of patterns that the network learned to recognize, we will try to generate images that maximize the sum of activations of particular channel of a particular convolutional layer of the neural network. The network we explore contains many convolutional layers, each of which outputs tens to hundreds of feature channels, so we have plenty of patterns to explore."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "LrucdvgyQgks",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "layers = [op.name for op in graph.get_operations() if op.type=='Conv2D' and 'import/' in op.name]\n",
        "feature_nums = [int(graph.get_tensor_by_name(name+':0').get_shape()[-1]) for name in layers]\n",
        "\n",
        "print('Number of layers', len(layers))\n",
        "print('Total number of feature channels:', sum(feature_nums))\n",
        "\n",
        "\n",
        "# Helper functions for TF Graph visualization\n",
        "\n",
        "def strip_consts(graph_def, max_const_size=32):\n",
        "    \"\"\"Strip large constant values from graph_def.\"\"\"\n",
        "    strip_def = tf.GraphDef()\n",
        "    for n0 in graph_def.node:\n",
        "        n = strip_def.node.add() \n",
        "        n.MergeFrom(n0)\n",
        "        if n.op == 'Const':\n",
        "            tensor = n.attr['value'].tensor\n",
        "            size = len(tensor.tensor_content)\n",
        "            if size \u003e max_const_size:\n",
        "                tensor.tensor_content = tf.compat.as_bytes(\"\u003cstripped %d bytes\u003e\"%size)\n",
        "    return strip_def\n",
        "  \n",
        "def rename_nodes(graph_def, rename_func):\n",
        "    res_def = tf.GraphDef()\n",
        "    for n0 in graph_def.node:\n",
        "        n = res_def.node.add() \n",
        "        n.MergeFrom(n0)\n",
        "        n.name = rename_func(n.name)\n",
        "        for i, s in enumerate(n.input):\n",
        "            n.input[i] = rename_func(s) if s[0]!='^' else '^'+rename_func(s[1:])\n",
        "    return res_def\n",
        "  \n",
        "def show_graph(graph_def, max_const_size=32):\n",
        "    \"\"\"Visualize TensorFlow graph.\"\"\"\n",
        "    if hasattr(graph_def, 'as_graph_def'):\n",
        "        graph_def = graph_def.as_graph_def()\n",
        "    strip_def = strip_consts(graph_def, max_const_size=max_const_size)\n",
        "    code = \"\"\"\n",
        "        \u003cscript\u003e\n",
        "          function load() {{\n",
        "            document.getElementById(\"{id}\").pbtxt = {data};\n",
        "          }}\n",
        "        \u003c/script\u003e\n",
        "        \u003clink rel=\"import\" href=\"https://tensorboard.appspot.com/tf-graph-basic.build.html\" onload=load()\u003e\n",
        "        \u003cdiv style=\"height:600px\"\u003e\n",
        "          \u003ctf-graph-basic id=\"{id}\"\u003e\u003c/tf-graph-basic\u003e\n",
        "        \u003c/div\u003e\n",
        "    \"\"\".format(data=repr(str(strip_def)), id='graph'+str(np.random.rand()))\n",
        "  \n",
        "    iframe = \"\"\"\n",
        "        \u003ciframe seamless style=\"width:800px;height:620px;border:0\" srcdoc=\"{}\"\u003e\u003c/iframe\u003e\n",
        "    \"\"\".format(code.replace('\"', '\u0026quot;'))\n",
        "    display(HTML(iframe))\n",
        "\n",
        "# Visualizing the network graph. Be sure expand the \"mixed\" nodes to see their \n",
        "# internal structure. We are going to visualize \"Conv2D\" nodes.\n",
        "tmp_def = rename_nodes(graph_def, lambda s:\"/\".join(s.split('_',1)))\n",
        "show_graph(tmp_def)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Nv2JqNLBhy1j"
      },
      "source": [
        "\u003ca id='naive'\u003e\u003c/a\u003e\n",
        "## Naive feature visualization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6LXaGEJkQgk4"
      },
      "source": [
        "Let's start with a naive way of visualizing these. Image-space gradient ascent!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "ZxC_XGGXQgk7",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "# Picking some internal layer. Note that we use outputs before applying the ReLU nonlinearity\n",
        "# to have non-zero gradients for features with negative initial activations.\n",
        "layer = 'mixed4d_3x3_bottleneck_pre_relu'\n",
        "channel = 139 # picking some feature channel to visualize\n",
        "\n",
        "# start with a gray image with a little noise\n",
        "img_noise = np.random.uniform(size=(224,224,3)) + 100.0\n",
        "\n",
        "def showarray(a, fmt='jpeg'):\n",
        "    a = np.uint8(np.clip(a, 0, 1)*255)\n",
        "    f = BytesIO()\n",
        "    PIL.Image.fromarray(a).save(f, fmt)\n",
        "    display(Image(data=f.getvalue()))\n",
        "    \n",
        "def visstd(a, s=0.1):\n",
        "    '''Normalize the image range for visualization'''\n",
        "    return (a-a.mean())/max(a.std(), 1e-4)*s + 0.5\n",
        "\n",
        "def T(layer):\n",
        "    '''Helper for getting layer output tensor'''\n",
        "    return graph.get_tensor_by_name(\"import/%s:0\"%layer)\n",
        "\n",
        "def render_naive(t_obj, img0=img_noise, iter_n=20, step=1.0):\n",
        "    t_score = tf.reduce_mean(t_obj) # defining the optimization objective\n",
        "    t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!\n",
        "    \n",
        "    img = img0.copy()\n",
        "    for i in range(iter_n):\n",
        "        g, score = sess.run([t_grad, t_score], {t_input:img})\n",
        "        # normalizing the gradient, so the same step size should work \n",
        "        g /= g.std()+1e-8         # for different layers and networks\n",
        "        img += g*step\n",
        "        print(score, end = ' ')\n",
        "    clear_output()\n",
        "    showarray(visstd(img))\n",
        "\n",
        "render_naive(T(layer)[:,:,:,channel])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ZroBKE5YiDsb"
      },
      "source": [
        "\u003ca id=\"multiscale\"\u003e\u003c/a\u003e\n",
        "## Multiscale image generation\n",
        "\n",
        "Looks like the network wants to show us something interesting! Let's help it. We are going to apply gradient ascent on multiple scales. Details formed on smaller scale will be upscaled and augmented with additional details on the next scale.\n",
        "\n",
        "With multiscale image generation it may be tempting to set the number of octaves to some high value to produce wallpaper-sized images. Storing network activations and backprop values will quickly run out of GPU memory in this case. There is a simple trick to avoid this: split the image into smaller tiles and compute each tile gradient independently. Applying random shifts to the image before every iteration helps avoid tile seams and improves the overall image quality."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "2iwWSOgsQglG",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "def tffunc(*argtypes):\n",
        "    '''Helper that transforms TF-graph generating function into a regular one.\n",
        "    See \"resize\" function below.\n",
        "    '''\n",
        "    placeholders = list(map(tf.placeholder, argtypes))\n",
        "    def wrap(f):\n",
        "        out = f(*placeholders)\n",
        "        def wrapper(*args, **kw):\n",
        "            return out.eval(dict(zip(placeholders, args)), session=kw.get('session'))\n",
        "        return wrapper\n",
        "    return wrap\n",
        "\n",
        "# Helper function that uses TF to resize an image\n",
        "def resize(img, size):\n",
        "    img = tf.expand_dims(img, 0)\n",
        "    return tf.image.resize_bilinear(img, size)[0,:,:,:]\n",
        "resize = tffunc(np.float32, np.int32)(resize)\n",
        "\n",
        "\n",
        "def calc_grad_tiled(img, t_grad, tile_size=512):\n",
        "    '''Compute the value of tensor t_grad over the image in a tiled way.\n",
        "    Random shifts are applied to the image to blur tile boundaries over \n",
        "    multiple iterations.'''\n",
        "    sz = tile_size\n",
        "    h, w = img.shape[:2]\n",
        "    sx, sy = np.random.randint(sz, size=2)\n",
        "    img_shift = np.roll(np.roll(img, sx, 1), sy, 0)\n",
        "    grad = np.zeros_like(img)\n",
        "    for y in range(0, max(h-sz//2, sz),sz):\n",
        "        for x in range(0, max(w-sz//2, sz),sz):\n",
        "            sub = img_shift[y:y+sz,x:x+sz]\n",
        "            g = sess.run(t_grad, {t_input:sub})\n",
        "            grad[y:y+sz,x:x+sz] = g\n",
        "    return np.roll(np.roll(grad, -sx, 1), -sy, 0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "GRCJdG8gQglN",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "def render_multiscale(t_obj, img0=img_noise, iter_n=10, step=1.0, octave_n=3, octave_scale=1.4):\n",
        "    t_score = tf.reduce_mean(t_obj) # defining the optimization objective\n",
        "    t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!\n",
        "    \n",
        "    img = img0.copy()\n",
        "    for octave in range(octave_n):\n",
        "        if octave\u003e0:\n",
        "            hw = np.float32(img.shape[:2])*octave_scale\n",
        "            img = resize(img, np.int32(hw))\n",
        "        for i in range(iter_n):\n",
        "            g = calc_grad_tiled(img, t_grad)\n",
        "            # normalizing the gradient, so the same step size should work \n",
        "            g /= g.std()+1e-8         # for different layers and networks\n",
        "            img += g*step\n",
        "            print('.', end = ' ')\n",
        "        clear_output()\n",
        "        showarray(visstd(img))\n",
        "\n",
        "render_multiscale(T(layer)[:,:,:,channel])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mDSZMtVYQglV"
      },
      "source": [
        "\u003ca id=\"laplacian\"\u003e\u003c/a\u003e\n",
        "## Laplacian Pyramid Gradient Normalization\n",
        "\n",
        "This looks better, but the resulting images mostly contain high frequencies. Can we improve it? One way is to add a smoothness prior into the optimization objective. This will effectively blur the image a little every iteration, suppressing the higher frequencies, so that the lower frequencies can catch up. This will require more iterations to produce a nice image. Why don't we just boost lower frequencies of the gradient instead? One way to achieve this is through the [Laplacian pyramid](https://en.wikipedia.org/wiki/Pyramid_%28image_processing%29#Laplacian_pyramid) decomposition. We call the resulting technique _Laplacian Pyramid Gradient Normalization_."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "Do3WpFSUQglX",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "k = np.float32([1,4,6,4,1])\n",
        "k = np.outer(k, k)\n",
        "k5x5 = k[:,:,None,None]/k.sum()*np.eye(3, dtype=np.float32)\n",
        "\n",
        "def lap_split(img):\n",
        "    '''Split the image into lo and hi frequency components'''\n",
        "    with tf.name_scope('split'):\n",
        "        lo = tf.nn.conv2d(img, k5x5, [1,2,2,1], 'SAME')\n",
        "        lo2 = tf.nn.conv2d_transpose(lo, k5x5*4, tf.shape(img), [1,2,2,1])\n",
        "        hi = img-lo2\n",
        "    return lo, hi\n",
        "\n",
        "def lap_split_n(img, n):\n",
        "    '''Build Laplacian pyramid with n splits'''\n",
        "    levels = []\n",
        "    for i in range(n):\n",
        "        img, hi = lap_split(img)\n",
        "        levels.append(hi)\n",
        "    levels.append(img)\n",
        "    return levels[::-1]\n",
        "\n",
        "def lap_merge(levels):\n",
        "    '''Merge Laplacian pyramid'''\n",
        "    img = levels[0]\n",
        "    for hi in levels[1:]:\n",
        "        with tf.name_scope('merge'):\n",
        "            img = tf.nn.conv2d_transpose(img, k5x5*4, tf.shape(hi), [1,2,2,1]) + hi\n",
        "    return img\n",
        "\n",
        "def normalize_std(img, eps=1e-10):\n",
        "    '''Normalize image by making its standard deviation = 1.0'''\n",
        "    with tf.name_scope('normalize'):\n",
        "        std = tf.sqrt(tf.reduce_mean(tf.square(img)))\n",
        "        return img/tf.maximum(std, eps)\n",
        "\n",
        "def lap_normalize(img, scale_n=4):\n",
        "    '''Perform the Laplacian pyramid normalization.'''\n",
        "    img = tf.expand_dims(img,0)\n",
        "    tlevels = lap_split_n(img, scale_n)\n",
        "    tlevels = list(map(normalize_std, tlevels))\n",
        "    out = lap_merge(tlevels)\n",
        "    return out[0,:,:,:]\n",
        "\n",
        "# Showing the lap_normalize graph with TensorBoard\n",
        "lap_graph = tf.Graph()\n",
        "with lap_graph.as_default():\n",
        "    lap_in = tf.placeholder(np.float32, name='lap_in')\n",
        "    lap_out = lap_normalize(lap_in)\n",
        "show_graph(lap_graph)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "zj8Ms-WqQgla",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "def render_lapnorm(t_obj, img0=img_noise, visfunc=visstd,\n",
        "                   iter_n=10, step=1.0, octave_n=3, octave_scale=1.4, lap_n=4):\n",
        "    t_score = tf.reduce_mean(t_obj) # defining the optimization objective\n",
        "    t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!\n",
        "    # build the laplacian normalization graph\n",
        "    lap_norm_func = tffunc(np.float32)(partial(lap_normalize, scale_n=lap_n))\n",
        "\n",
        "    img = img0.copy()\n",
        "    for octave in range(octave_n):\n",
        "        if octave\u003e0:\n",
        "            hw = np.float32(img.shape[:2])*octave_scale\n",
        "            img = resize(img, np.int32(hw))\n",
        "        for i in range(iter_n):\n",
        "            g = calc_grad_tiled(img, t_grad)\n",
        "            g = lap_norm_func(g)\n",
        "            img += g*step\n",
        "            print('.', end = ' ')\n",
        "        clear_output()\n",
        "        showarray(visfunc(img))\n",
        "\n",
        "render_lapnorm(T(layer)[:,:,:,channel])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YzXJUF2lQgln"
      },
      "source": [
        "\u003ca id=\"playing\"\u003e\u003c/a\u003e\n",
        "## Playing with feature visualizations\n",
        "\n",
        "We got a nice smooth image using only 10 iterations per octave. In case of running on GPU this takes just a few seconds. Let's try to visualize another channel from the same layer. The network can generate wide diversity of patterns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "a6jfiWqZQglq",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "render_lapnorm(T(layer)[:,:,:,65])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ka6RyOMEnrB5"
      },
      "source": [
        "Lower layers produce features of lower complexity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "KYOtrJxMnlws",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "render_lapnorm(T('mixed3b_1x1_pre_relu')[:,:,:,101])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wuP8a4FlQglx"
      },
      "source": [
        "There are many interesting things one may try. For example, optimizing a linear combination of features often gives a \"mixture\" pattern."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "ozN-nH2yQgl0",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "render_lapnorm(T(layer)[:,:,:,65]+T(layer)[:,:,:,139], octave_n=4)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lcPe-ZMv0dYR"
      },
      "source": [
        "\u003ca id=\"deepdream\"\u003e\u003c/a\u003e\n",
        "## DeepDream\n",
        "\n",
        "Now let's reproduce the [DeepDream algorithm](https://github.com/google/deepdream/blob/master/dream.ipynb) with TensorFlow. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "qM2U_96hyUwN",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "def render_deepdream(t_obj, img0=img_noise,\n",
        "                     iter_n=10, step=1.5, octave_n=4, octave_scale=1.4):\n",
        "    t_score = tf.reduce_mean(t_obj) # defining the optimization objective\n",
        "    t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!\n",
        "\n",
        "    # split the image into a number of octaves\n",
        "    img = img0\n",
        "    octaves = []\n",
        "    for i in range(octave_n-1):\n",
        "        hw = img.shape[:2]\n",
        "        lo = resize(img, np.int32(np.float32(hw)/octave_scale))\n",
        "        hi = img-resize(lo, hw)\n",
        "        img = lo\n",
        "        octaves.append(hi)\n",
        "    \n",
        "    # generate details octave by octave\n",
        "    for octave in range(octave_n):\n",
        "        if octave\u003e0:\n",
        "            hi = octaves[-octave]\n",
        "            img = resize(img, hi.shape[:2])+hi\n",
        "        for i in range(iter_n):\n",
        "            g = calc_grad_tiled(img, t_grad)\n",
        "            img += g*(step / (np.abs(g).mean()+1e-7))\n",
        "            print('.',end = ' ')\n",
        "        clear_output()\n",
        "        showarray(img/255.0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "EuvInTo8n2Hk"
      },
      "source": [
        "Let's load some image and populate it with DogSlugs (in case you've missed them)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "exYchiCWmR4r"
      },
      "outputs": [],
      "source": [
        "img_path = tf.keras.utils.get_file(\"pilatus800.jpg\",\"https://storage.googleapis.com/download.tensorflow.org/example_images/pilatus800.jpg\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "M9_vOh_2Qgl-",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "img0 = PIL.Image.open(img_path)\n",
        "img0 = np.float32(img0)\n",
        "showarray(img0/255.0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "k0oggbGEeC3U",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "render_deepdream(tf.square(T('mixed4c')), img0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IJzvhEFxpB7E"
      },
      "source": [
        "Note that results can differ from the [Caffe](https://github.com/BVLC/caffe)'s implementation, as we are using an independently trained network. Still, the network seems to like dogs and animal-like features due to the nature of the ImageNet dataset.\n",
        "\n",
        "Using an arbitrary optimization objective still works:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "both",
        "colab": {},
        "colab_type": "code",
        "id": "4GexZuwJdDmu",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "render_deepdream(T(layer)[:,:,:,139], img0)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mYsY6_Ngpfwl"
      },
      "source": [
        "Don't hesitate to use higher resolution inputs (also increase the number of octaves)! Here is an [example](https://storage.googleapis.com/deepdream/pilatus_flowers.jpg) of running the flower dream over the bigger image."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mENNVQd3eD-h"
      },
      "source": [
        "We hope that the visualization tricks described here may be helpful for analyzing representations learned by neural networks or find their use in various artistic applications."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "deepdream.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true,
      "version": "0.3.2"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
